Methods and systems for replicating an expandable storage volume

ABSTRACT

Machine implemented method for generating and accessing a point in time copy of an expandable storage volume in a network storage system is provided. The expandable storage volume includes a namespace for storing information for accessing data objects stored at a plurality of data constituent volumes managed by a plurality of nodes of the storage system. The method includes initiating a logical fence first for the namespace and then for the data constituent volumes for generating the point in time copy of the expandable storage volume. The logical fence filters out any information written after the fence is initiated from the point in time copy of the expandable storage volume.

CROSS REFERENCE TO RELATED APPLICATION

This patent application is related to U.S. patent application Ser. No.13/433,170, entitled “METHODS AND SYSTEMS FOR REPLICATING AN EXPANDABLESTORAGE VOLUME”, filed on Mar. 28, 2012, the disclosure of which isincorporated herein by its entirety.

TECHNICAL FIELD

At least one embodiment of the present invention pertains to storagesystems, and more particularly, to methods and systems for replicatingstorage volumes.

BACKGROUND

Network based storage, or simply “network storage”, is a common approachto backing up data, making large amounts of data accessible to multipleusers, and other purposes. In a network storage environment, a storageserver (or storage system) makes data available to client (also referredto as “host”) systems by presenting or exporting to clients one or morelogical data containers. There are various forms of network storage,including network attached storage (NAS) and storage area network (SAN).In a NAS context, a storage server services file-level requests fromclients, whereas in a SAN context a storage server services block-levelrequests. Some storage servers are capable of servicing both file-levelrequests and block-level requests.

There are several trends that are relevant to network storagetechnology. The first is that the amount of data being stored within atypical enterprise is increasing from year to year. Second, there arenow multiple mechanisms (or protocols) by which a user may wish toaccess data stored in network storage system. For example, consider acase where a user wishes to access a document stored at a particularlocation in a network storage system. The user may use a network filesystem (NFS) protocol to access the document over a local area networkin a manner similar to how local storage is accessed. The user may alsouse an HTTP protocol to access a document over a wide area network suchas an Internet network. Traditional storage systems use a differentstorage mechanism (e.g., a different file system) for presenting dataover each such protocol. Accordingly, traditional network storagesystems do not allow the same stored data to be accessed concurrentlyover multiple different protocols at the same level of a protocol stack.

In addition, network storage systems presently are constrained in theway they allow a user to store or navigate data. Consider, for example,a photo that is stored under a given path name, such as“/home/eng/myname/office.jpeg”. In a traditional network storage system,this path name maps to a specific storage volume and a specific filelocation (e.g., an inode number). Thus, a path name of a file (e.g., aphoto) is closely tied to the file's storage location. In other words,the physical storage location of the file is determined by the path nameof the file. Accordingly, in traditional storage systems, the path nameof the file needs to be updated every time the physical storage locationof the file changes (e.g., when the file is transferred to a differentstorage volume). This characteristic significantly limits theflexibility of the system.

Continuous efforts are being made to provide a flexible, expandablestorage system, where data objects may be stored and replicated acrossstorage volumes that may be managed by different storage system nodes.

SUMMARY

In one embodiment, a machine implemented method for generating andaccessing a point in time copy (or snapshot) of an expandable storagevolume in a network storage system is provided. The expandable storagevolume includes a namespace for storing information for accessing dataobjects stored at a plurality of data constituent volumes managed by aplurality of nodes of the storage system. The method includes initiatinga logical fence first for the namespace and then for the dataconstituent volumes for generating the point in time copy of theexpandable storage volume. The logical fence filters out any informationwritten after the fence is initiated from the point in time copy of theexpandable storage volume. After the point in time copy is generated,information regarding the point in time copy of the expandable storagevolume is stored as a data structure such that the point in time copy ofthe expandable storage volume is presented to the client as a singlelogical entity for accessing a point in time copy of the namespace and apoint in time copy of the data constituent volumes.

In another embodiment, a machine implemented method for a snapshot of anexpandable storage volume in a network storage system is provided. Theexpandable storage volume includes a namespace for storing informationfor accessing data objects stored at a data constituent volume. Themethod includes initiating a logical fence first for the namespace andthen for the data constituent volume for generating the snapshot of theexpandable storage volume; storing information regarding the snapshot ofthe expandable storage volume in a data structure such that the snapshotof the expandable storage volume is presented to the client as a singleentity for accessing a snapshot of the namespace and a snapshot of thedata constituent volume; receiving an object for accessing the snapshotof the expandable storage volume, the object includes an identifieridentifying the snapshot of the namespace and an identifier identifyingthe namespace.

The method further includes retrieving an identifier identifying thesnapshot of the data constituent volume from the data structure; andretrieving data associated with the snapshot of the data constituentvolume based on the identifier of the snapshot of the data constituentvolume.

In yet another embodiment, a machine implemented method for generating asnapshot of an expandable storage volume in a network storage system isprovided. The expandable storage volume includes a namespace for storinginformation for accessing data objects stored at a data constituentvolume. The method includes retrieving identifier information of thenamespace and the data constituent volume from a volume data structureusing an identifier of the expandable storage volume; initiating alogical fence first on the namespace and then on the data constituentfor generating the snapshot of the expandable storage volume and afterthe expandable storage volume is replicated, the logical fence isremoved in an order opposite to an order in which the fence wasinitiated; and storing information regarding the snapshot of theexpandable storage volume as a data structure such that the snapshot ofthe expandable storage volume is presented to the client as a singleentity for accessing a snapshot of the namespace and a snapshot of thedata constituent volume.

In another embodiment, a machine implemented method for accessing asnapshot of an expandable storage volume in a network storage system isprovided. The expandable storage volume includes a namespace for storinginformation for accessing data objects stored at a data constituentvolume. The method includes receiving an object for accessing thesnapshot of the expandable storage volume, where the object includes anidentifier identifying the snapshot of the namespace and an identifieridentifying the namespace; retrieving an identifier identifying asnapshot of the data constituent volume from a data structure used tostore information regarding the snapshot of the expandable storagevolume for presenting the snapshot of the expandable storage volume as asingle logical entity; and retrieving data associated with the snapshotof the data constituent volume based on the identifier of the snapshotof the data constituent volume.

This brief summary has been provided so that the nature of thisdisclosure may be understood quickly. A more complete understanding ofthe disclosure can be obtained by reference to the following detaileddescription of the various embodiments thereof in connection with theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features and other features will now be described withreference to the drawings of the various embodiments. In the drawings,the same components have the same reference numerals. The illustratedembodiments are intended to illustrate, but not to limit the presentdisclosure. The drawings include the following Figures:

FIG. 1 illustrates a network storage environment, according to oneembodiment.

FIG. 2 illustrates a clustered network storage environment, according toone embodiment.

FIG. 3 is a high-level block diagram showing an example of the hardwarearchitecture of a storage controller that can implement one or morestorage server nodes, according to one embodiment.

FIG. 4 illustrates an example of a storage operating system of a storageserver node, used according to one embodiment.

FIG. 5A illustrates the overall architecture of a content repositoryaccording to one embodiment.

FIG. 5B illustrates a content repository that can be implemented in theclustered architecture of FIGS. 2 through 4.

FIG. 5C illustrates a multilevel object handle, according to oneembodiment.

FIG. 5D illustrates a mechanism for a storage system to introduce alayer of separation between a directory entry of a data object and thephysical location where the data object is stored, according to oneembodiment.

FIG. 5E illustrates a mechanism that allows a storage system tointroduce a layer of separation between the directory entry of the dataobject and the physical location of the data object by including aglobal object ID within the directory entry.

FIG. 6 shows a block diagram of an expandable storage volume that isreplicated according to one embodiment.

FIG. 7A shows a block diagram of a system for replicating an expandablestorage volume, according to one embodiment.

FIG. 7B shows a volume data structure used according to one embodiment.

FIG. 7C shows an example of using a consistency group for replicating astorage volume, according to one embodiment.

FIG. 7D shows a snapshot data structure for replicating an expandablestorage volume, according to one embodiment.

FIG. 8 is a process flow for replicating an expandable storage volume,according to one embodiment.

FIG. 9A is a process flow diagram for accessing a replicated version ofan expandable storage volume, according to one embodiment.

FIG. 9B shows an example of implementing the process of FIG. 9A,according to one embodiment.

DETAILED DESCRIPTION

As a preliminary note, the terms “component”, “module”, “system,” andthe like as used in this disclosure are intended to refer to acomputer-related entity, either software-executing general purposeprocessor, hardware, firmware and a combination thereof. For example, acomponent may be, but is not limited to being, a process running on aprocessor, a processor, an object, an executable, a thread of execution,a program, and/or a computer.

By way of illustration, both an application running on a server and theserver can be a component. One or more components may reside within aprocess and/or thread of execution and a component may be localized onone computer and/or distributed between two or more computers. Also,these components can execute from various computer readable media havingvarious data structures stored thereon. The components may communicatevia local and/or remote processes such as in accordance with a signalhaving one or more data packets (e.g., data from one componentinteracting with another component in a local system, distributedsystem, and/or across a network such as the Internet with other systemsvia the signal).

Computer executable components can be stored, for example, onnon-transitory computer readable media including, but not limited to, anASIC (application specific integrated circuit), CD (compact disc), DVD(digital video disk), ROM (read only memory), floppy disk, hard disk,EEPROM (electrically erasable programmable read only memory), memorystick, flash memory device or any other non-volatile memory device, orany other storage device, in accordance with the claimed subject matter.

In one embodiment, a machine implemented method and system forgenerating and accessing a point in time copy of an expandable storagevolume in a network storage system is provided. The expandable storagevolume includes a namespace for storing information for accessing dataobjects stored at a plurality of data constituent volumes managed by aplurality of nodes of the storage system. The method includes initiatinga logical fence first for the namespace and then for the dataconstituent volumes for generating the point in time copy of theexpandable storage volume. The logical fence filters out any informationwritten after the fence is initiated from the point in time copy of theexpandable storage volume. After the point in time copy is generated,information regarding the point in time copy of the expandable storagevolume is stored as a data structure such that the point in time copy ofthe expandable storage volume is presented to the client as a singlelogical entity for accessing a point in time copy of the namespace and apoint in time copy of the data constituent volumes.

System Environment:

FIGS. 1 and 2 show, at different levels of detail, storage environmentconfigurations in which the techniques introduced here can beimplemented. Clients are presented with an expandable volume having aplurality of storage volumes that can be managed by a plurality ofnodes. The expandable volume is replicated, for example, by takingsnapshots of each storage volume. The techniques described below indetail present the plurality of snapshots as a single object (or logicalentity) to a client. The client is able to access the snapshots of thevarious volumes using the single object, as described below in detail.

Referring to FIG. 1, a network data storage environment 100 is shown.The storage environment 100 includes a plurality of client systems104.1-104.N, a storage server system 102, and a network 106 connectingthe client systems 104.1-104.N and the storage server system 102. Asshown in FIG. 1, the storage server system 102 includes at least onestorage server 108, a switching fabric 110, and a number of mass storagedevices 112 within a mass storage subsystem 114, such as conventionalmagnetic disks, optical disks such as CD-ROM or DVD based storage,magneto-optical (MO) storage, flash memory storage device or any othertype of non-volatile storage devices suitable for storing structured orunstructured data. The examples disclosed herein may reference a storagedevice as a “disk” but the adaptive embodiments disclosed herein are notlimited to disks or any particular type of storage media/device, in themass storage subsystem 114.

The storage server (or servers) 108 may be, for example, one of thestorage server products available from NetApp, Inc., the assignee of thepresent application. The client systems 104.1-104.N may access thestorage server 108 via network 106, which can be a packet-switchednetwork, for example, a local area network (LAN), wide area network(WAN) or any other type of network.

The storage server 108 maybe connected to the storage devices 112 viathe switching fabric 110, which can be a fiber distributed datainterface (FDDI) network, for example. It is noted that, within thenetwork data storage environment, any other suitable numbers of storageservers and/or mass storage devices, and/or any other suitable networktechnologies, may be employed. While FIG. 1 implies, in someembodiments, a fully connected switching fabric 110 where storageservers can see all storage devices, it is understood that such aconnected topology is not required. In some embodiments, the storagedevices can be directly connected to the storage servers such that notwo storage servers see a given storage device.

The storage server 108 can make some or all of the storage space on thestorage devices 112 available to the client systems 104.1-104.N in aconventional manner. For example, each storage device 112 can beimplemented as an individual disk, multiple disks (e.g., a RAID group)or any other suitable mass storage device(s). The storage server 108 cancommunicate with the client systems 104.1-104.N according to well-knownprotocols, such as the Network File System (NFS) protocol or the CommonInternet File System (CIFS) protocol, to make data stored at storagedevices 112 available to users and/or application programs.

The storage server 108 can present or export data stored at storagedevice 112 as volumes (also referred to herein as storage volumes) toeach of the client systems 104.1-104.N. A “volume” is an abstraction ofphysical storage, combining one or more physical mass storage devices(e.g., disks) or parts thereof into a single logical storage object (thevolume), and which is managed as a single administrative unit, such as asingle file system. A “file system” is a structured (e.g., hierarchical)set of stored logical containers of data (e.g., volumes, logical unitnumbers (LUNs), directories, files). Note that a “file system” does nothave to include or be based on “files” per se as its units of datastorage.

Various functions and configuration settings of the storage server 108and the mass storage subsystem 114 can be controlled from a managementconsole 116 coupled to the network 106.

FIG. 2 depicts a cluster based storage environment 200 having aplurality of server nodes, according to one embodiment. In the clusterbased storage environment 200, clients may be presented with anexpandable storage volume (for example, an Infinite volume 600 describedbelow with respect to FIG. 6) having a plurality of storage volumes thatare managed by different server nodes. The various storage volumes arereplicated using the techniques described below in detail.

The storage environment 200 includes a plurality of client systems 204(204.1-204.M), a clustered storage system 202, and a network 206connecting the client systems 204 and the clustered storage serversystem 202. As shown in FIG. 2, the clustered storage server system 202includes a plurality of server nodes (may also be referred to as“nodes”) 208.1-208.N (208), a cluster switching fabric 210, and aplurality of mass storage devices 212 (212.1-212.N), similar to storagedevices 112 (FIG. 1). Note that more than one mass storage device 212can be associated with each node 208.

Each of the nodes 208 is configured to include several modules,including an N-module 214, a D-module 216, and an M-host 218 (each ofwhich can be implemented by using a separate processor executablemodule) and an instance of a replicated database (RDB) 220.Specifically, node 208.1 includes an N-module 214.1, a D-module 216.1,and an M-host 218.1; node 208.N includes an N-module 214.N, a D-module216.N, and an M-host 218.N; and so forth. The N-modules 214.1-214.Ninclude functionality that enables nodes 208.1-208.N, respectively, toconnect to one or more of the client systems 204 over the network 206,while the D-modules 216.1-216.N provide access to the data stored atstorage devices 212.1-212.N, respectively. The M-hosts 218 providemanagement functions for the clustered storage server system 202including a system for replicating the Infinite Volume 600 describedbelow in detail. Accordingly, each of the server nodes 208 in theclustered storage server arrangement provides the functionality of astorage server.

In one embodiment RDB 220 is a database that is replicated throughoutthe cluster, i.e., each node 208 includes an instance of the RDB 220.The various instances of the RDB 220 are updated regularly to bring theminto synchronization with each other. The RDB 220 provides cluster-widestorage for information used by nodes 208, including a volume locationdatabase (VLDB) (not shown). The VLDB is a database that indicates thelocation within the cluster of each volume in the cluster (i.e., theowning D-module 216 for each volume) and is used by the N-modules 214 toidentify the appropriate D-module 216 for any given volume to whichaccess is requested.

A switched virtualization layer including a plurality of virtualinterfaces (VIFs) 222 is provided between the respective N-modules214.1-214.N and the client systems 204.1-204.M, allowing the storage212.1-212.N associated with the nodes 208.1-208.N to be presented to theclient systems as a single shared storage pool.

The clustered storage system 202 can be organized into any suitablenumber of virtual servers (also referred to as “vservers”), in whicheach vserver represents a single storage system namespace with separatenetwork access. Each vserver has a user domain and a security domainthat are separate from the user and security domains of other vservers.Moreover, each vserver is associated with one or more VIFs 222 and canspan one or more physical nodes, each of which can hold one or more VIFs222 and storage associated with one or more vservers. Client systems canaccess the data on a vserver from any node of the clustered system, butonly through the VIFs 222 associated with that vserver. It is noteworthythat the embodiments described herein are not limited to the use ofvservers.

The nodes 208 are interconnected by a cluster switching fabric 210,which can be embodied as a Gigabit Ethernet switch, for example. TheN-modules 214 and D-modules 216 cooperate to provide highly-scalable,distributed storage system architecture of a clustered computingenvironment implementing exemplary embodiments of the present invention.Note that while there is shown an equal number of N-modules andD-modules in FIG. 2, there may be differing numbers of N-modules and/orD-modules in accordance with various embodiments of the techniquedescribed here. For example, there need not be a one-to-onecorrespondence between the N-modules and D-modules. As such, thedescription of a node 208 comprising one N-module and one D-moduleshould be understood to be illustrative only.

FIG. 3 is a diagram illustrating an example for implementing one or moreof the storage server nodes 208 as a storage controller 300. The storagecontroller 300 executes some or all of the processor executable processsteps that are described below in detail. In one embodiment, the storagecontroller 300 includes a processor subsystem that includes one or moreprocessors 302. Processor 302 may be or may include, one or moreprogrammable general-purpose or special-purpose microprocessors, digitalsignal processors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), or thelike, or a combination of such hardware based devices.

The storage controller 300 further includes a memory 304, a networkadapter 310, a cluster access adapter 312 and a storage adapter 314, allinterconnected by an interconnect 308. Interconnect 308 may include, forexample, a system bus, a Peripheral Component Interconnect (PCI) bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), oran Institute of Electrical and Electronics Engineers (IEEE) standard1394 bus (sometimes referred to as “Firewire”) or any other system.

The cluster access adapter 312 includes a plurality of ports adapted tocouple the node 208 to other nodes 208 of the cluster. In theillustrated embodiment, Ethernet is used as the clustering protocol andinterconnect media, although other types of protocols and interconnectsmay be utilized within the cluster architecture described herein. Inalternative embodiments where the N-modules and D-modules areimplemented on separate storage systems or computers, the cluster accessadapter 312 is utilized by the N-module 214 and/or D-module 216 forcommunicating with other N-modules and/or D-modules of the cluster.

The storage controller 300 can be embodied as a single- ormulti-processor storage system executing a storage operating system 306that preferably implements a high-level module, such as a storagemanager, to logically organize the information as a hierarchicalstructure of named directories, files and special types of files calledvirtual disks (hereinafter generally “blocks”) at the storage devices.Illustratively, one processor 302 can execute the functions of theN-module 214 on the node 208 while another processor 302 executes thefunctions of the D-module 216.

The memory 304 illustratively comprises storage locations that areaddressable by the processors and adapters 310, 312, 314 for storingprocessor executable code and data structures associated with thepresent disclosure. The processor 302 and adapters may, in turn,comprise processing elements and/or logic circuitry configured toexecute the software code and manipulate the data structures. Thestorage operating system 306, portions of which is typically resident inmemory and executed by the processors(s) 302, functionally organizes thestorage controller 300 by (among other things) configuring theprocessor(s) 302 to invoke storage operations in support of the storageservice provided by the node 208. It will be apparent to those skilledin the art that other processing and memory implementations, includingvarious computer readable storage media, may be used for storing andexecuting program instructions pertaining to the technique introducedhere.

The network adapter 310 includes a plurality of ports to couple thestorage controller 300 to one or more clients 204 over point-to-pointlinks, wide area networks, virtual private networks implemented over apublic network (Internet) or a shared local area network. The networkadapter 310 thus can include the mechanical, electrical and signalingcircuitry needed to connect the storage controller 300 to the network206. Illustratively, the network 206 can be embodied as an Ethernetnetwork or a Fibre Channel (FC) network. Each client 204 can communicatewith the node 208 over the network 206 by exchanging discrete frames orpackets of data according to pre-defined protocols, such as TCP/IP.

The storage adapter 314 cooperates with the storage operating system 306to access information requested by the clients 204. The information maybe stored on any type of attached array of writable storage media, suchas magnetic disk or tape, optical disk (e.g., CD-ROM or DVD), flashmemory, solid-state disk (SSD), electronic random access memory (RAM),micro-electro mechanical and/or any other similar media adapted to storeinformation, including data and parity information. However, asillustratively described herein, the information is stored on storagedevices 212. The storage adapter 314 includes a plurality of portshaving input/output (I/O) interface circuitry that couples to the disksover an I/O interconnect arrangement, such as a conventionalhigh-performance, Fibre Channel (FC) link topology.

Storage of information on storage devices 212 can be implemented as oneor more storage volumes that include a collection of physical storagedisks cooperating to define an overall logical arrangement of volumeblock number (VBN) space on the volume(s). The storage devices 212 canbe organized as a RAID group. One or more RAID groups together form anaggregate. An aggregate can contain one or more volumes/file systems.

The storage operating system 306 facilitates clients access to datastored on the storage devices 212. In certain embodiments, the storageoperating system 306 implements a write-anywhere file system thatcooperates with one or more virtualization modules to “virtualize” thestorage space provided by storage devices 212. In certain embodiments, astorage manager 406 (FIG. 4) logically organizes the information as ahierarchical structure of named directories and files on the storagedevices 212. Each “on-disk” file may be implemented as set of diskblocks configured to store information, such as data, whereas thedirectory may be implemented as a specially formatted file in whichnames and links to other files and directories are stored. Thevirtualization module(s) allow the storage manager 406 to furtherlogically organize information as a hierarchical structure of blocks onthe disks that are exported as named logical unit numbers (LUNs).

In the illustrative embodiment, the storage operating system 306 is aversion of the Data ONTAP® operating system available from NetApp, Inc.and the storage manager 406 implements the Write Anywhere File Layout(WAFL®) file system. However, other storage operating systems arecapable of being enhanced or created for use in accordance with theprinciples described herein.

FIG. 4 is a diagram illustrating an example of storage operating system306 that can be used with the replication techniques introduced here.The storage operating system 306 may be used to maintain various datastructures for replicating storage volumes and providing access toreplicated storage volumes, as described below in more detail withrespect to FIGS. 6-9B.

In the illustrated embodiment the storage operating system 306 includesmultiple functional layers organized to form an integrated networkprotocol stack or, more generally, a multi-protocol engine 416 thatprovides data paths for clients to access information stored on the nodeusing block and file access protocols. The multiprotocol engine 416 incombination with underlying processing hardware also forms the N-module214. The multi-protocol engine 416 includes a network access layer 404which includes one or more network drivers that implement one or morelower-level protocols to enable the processing system to communicateover the network 206, such as Ethernet, Internet Protocol (IP),Transport Control Protocol/Internet Protocol (TCP/IP), Fibre ChannelProtocol (FCP) and/or User Datagram Protocol/Internet Protocol (UDP/IP).The multiprotocol engine 416 also includes a protocol layer 402 whichimplements various higher-level network protocols, such as NFS, CIFS,Hypertext Transfer Protocol (HTTP), Internet small computer systeminterface (iSCSI), etc. Further, the multiprotocol engine 416 includes acluster fabric (CF) interface module 400A which implements intra-clustercommunication with D-modules and with other N-modules.

In addition, the storage operating system 306 includes a set of layersorganized to form a backend server 412 that provides data paths foraccessing information stored on the storage devices 212 of the node 208.The backend server 412 in combination with underlying processinghardware also forms the D-module 216. To that end, the backend server412 includes a storage manager module 406 that manages any number ofstorage volumes, a RAID system module 408 and a storage driver systemmodule 410.

The storage manager 406 primarily manages a file system (or multiplefile systems) and serves client-initiated read and write requests. TheRAID system 408 manages the storage and retrieval of information to andfrom the volumes/disks in accordance with a RAID redundancy protocol,such as RAID-4, RAID-5, or RAID-DP, while the storage driver system 410implements a disk access protocol such as SCSI protocol or FCP.

The backend server 412 also includes a CF interface module 400B toimplement intra-cluster communication 414 with N-modules and/or otherD-modules. The CF interface modules 400A and 400B can cooperate toprovide a single file system image across the D-modules 216 in thecluster. Thus, any network port of an N-module 214 that receives aclient request can access any data container within the single filesystem image located on any D-module 216 of the cluster.

The CF interface modules 400A/400B implement the CF protocol tocommunicate file system commands among the modules of cluster over thecluster switching fabric 210 (FIG. 2). Such communication can beeffected by a D-module exposing a CF application programming interface(API) to which an N-module (or another D-module) issues calls. To thatend, a CF interface module can be organized as a CF encoder/decoder. TheCF encoder of, e.g., CF interface 400A on N-module 214 can encapsulate aCF message as (i) a local procedure call (LPC) when communicating a filesystem command to a D-module 216 residing on the same node or (ii) aremote procedure call (RPC) when communicating the command to a D-moduleresiding on a remote node of the cluster. In either case, the CF decoderof CF interface 400B on D-module 216 de-encapsulates the CF message andprocesses the file system command.

In operation of a node 208, a request from a client 204 is forwarded asa packet over the network 206 and onto the node 208, where it isreceived at the network adapter 310 (FIG. 3). A network driver of layer404 processes the packet and, if appropriate, passes it on to a networkprotocol and file access layer for additional processing prior toforwarding to the storage manager 406. At that point, the storagemanager 406 generates operations to load (retrieve) the requested datafrom storage device 212 if it is not resident in memory 304. If theinformation is not in memory 304, the storage manager 406 indexes into ametadata file to access an appropriate entry and retrieve a logicalvirtual block number (VBN). The storage manager 406 then passes amessage structure including the logical VBN to the RAID system 408; thelogical VBN is mapped to a disk identifier and disk block number (DBN)and sent to an appropriate driver (e.g., SCSI) of the storage driversystem 410. The storage driver accesses the DBN from the specifiedstorage device 212 and loads the requested data block(s) in memory forprocessing by the node. Upon completion of the request, the node (andoperating system) returns a reply to the client 204 over the network206.

The data request/response “path” through the storage operating system306 as described above can be implemented in general-purposeprogrammable hardware executing the storage operating system 306 assoftware or firmware. Alternatively, it can be implemented at leastpartially in specially designed hardware. That is, in an alternateembodiment of the invention, some or all of the storage operating system306 is implemented as logic circuitry embodied within a fieldprogrammable gate array (FPGA) or an application specific integratedcircuit (ASIC), for example.

The N-module 214 and D-module 216 each can be implemented as processinghardware configured by separately-scheduled processes of storageoperating system 306; however, in an alternate embodiment, the modulesmay be implemented as processing hardware configured by code within asingle operating system process. Communication between an N-module 214and a D-module 216 is thus illustratively effected through the use ofmessage passing between the modules although, in the case of remotecommunication between an N-module and D-module of different nodes, suchmessage passing occurs over the cluster switching fabric 210. A knownmessage-passing mechanism provided by the storage operating system totransfer information between modules (processes) is the Inter ProcessCommunication (IPC) mechanism. The protocol used with the IPC mechanismis illustratively a generic file and/or block-based “agnostic” CFprotocol that comprises a collection of methods/functions constituting aCF API.

Overview of Content Repository:

The techniques introduced here generally relate to a content repositoryimplemented in a network storage server system 202 such as describedabove. FIG. 5A illustrates the overall architecture of the contentrepository according to one embodiment. The content repository includesa distributed object store 518, an object location subsystem (OLS) 516,a presentation layer 502, and a management subsystem 514. Normally therewill be a single instance of each of these components in the overallcontent repository, and each of these components can be implemented inany one server node 208 or distributed across two or more server nodes208. The functional elements of each of these units (i.e., the OLS 516,presentation layer 502 and management subsystem 514) can be implementedby specially designed circuitry, or by programmable circuitry programmedwith software and/or firmware, or a combination thereof. The datastorage elements of these units can be implemented using any known orconvenient form or forms of data storage device.

The distributed object store 518 provides the actual data storage forthe data objects in the server system 202 and includes multiple dataconstituent volumes (may interchangeably be referred to as distinctsingle-node object stores 520). A “single-node” object store or dataconstituent volume is an object store that is implemented entirelywithin one node. Each data constituent volume 520 is a logical(non-physical) container of data, such as a data constituent volume or alogical unit (LUN). Some or all of the data constituent volumes 520 thatmake up the distributed object store 518 can be implemented in separateserver nodes 208. Alternatively, all of the data constituent volumes 520that make up the distributed object store 518 can be implemented in thesame server node. Any given server node 208 can access multiple dataconstituent volumes 520 and can include multiple data constituentvolumes 520.

The distributed object store 518 provides location-independentaddressing of data objects (i.e., data objects can be moved among dataconstituent volumes 520 without changing the data objects' addressing),with the ability to span the object address space across other similarsystems spread over geographic distances. Note that the distributedobject store 518 has no namespace; the namespace for the server system202 is provided by the presentation layer 502.

The term “namespace” as used herein refers to a virtual hierarchicalcollection of unique volume names or identifiers and directory paths tothe volumes, in which each volume represents a virtualized containerstoring a portion of the namespace descending from a single rootdirectory. For example, each volume associated with a namespace can beconfigured to store one or more data containers, scripts, wordprocessing documents, executable programs and others.

The presentation layer 502 provides access to the distributed objectstore 518. It is generated by at least one presentation module 500(i.e., it may be generated collectively by multiple presentation modules500, one in each multiple server nodes 208). The presentation module 500can be in the form of specially designed circuitry, or programmablecircuitry programmed with software and/or firmware, or a combinationthereof.

The presentation layer 502 receives client requests, translates theminto an internal protocol and sends them to the appropriate D-module216. The presentation layer 502 provides two or more independentinterfaces for accessing stored data, e.g., a conventional NAS interface504 and a Web Service interface 508. The NAS interface 504 allows accessto the object store 518 via one or more conventional NAS protocols, suchas NFS and/or CIFS. Thus, the NAS interface 504 provides a filesystem-like interface to the content repository.

The Web Service interface 508 allows access to data stored in the objectstore 518 via either “named object access” or “raw object access” (alsocalled “flat object access”). Named object access uses a namespace(e.g., a file system-like directory-tree interface for accessing dataobjects), as does NAS access; whereas raw object access usessystem-generated global object IDs to access data objects, as describedfurther below. The Web Service interface 508 allows access to the objectstore 518 via Web Service (as defined by the W3C), using for example, aprotocol such as Simple Object Access Protocol (SOAP) or a RESTful(REpresentational State Transfer-ful) protocol, over HTTP.

The presentation layer 502 further provides at least one namespace 506(may also be referred to as namespace volume) for accessing data via theNAS interface or the Web Service interface. In one embodiment thisincludes a Portable Operating System Interface (POSIX) namespace. TheNAS interface 504 allows access to data stored in the object store 518via the namespace(s) 506. The Web Service interface 508 allows access todata stored in the object store 518 via either the namespace(s) 506 (byusing named object access) or without using the namespace(s) 506 (byusing “raw object access”). Thus, the Web Service interface 508 allowseither named object access or raw object access; and while named objectaccess is accomplished using a namespace 506, raw object access is not.Access by the presentation layer 502 to the object store 518 is viaeither a “fast path” 524 or a “slow path” 522, as discussed furtherbelow.

The function of the OLS 516 is to store and provide valid location IDs(and other information, such as policy IDs) of data objects, based ontheir global object IDs (these parameters are discussed further below).This is done, for example, when a client 204 requests access to a dataobject by using only the global object ID instead of a complete objecthandle including the location ID, or when the location ID within anobject handle is no longer valid (e.g., because the target data objecthas been moved). Note that the system 202 thereby provides two distinctpaths for accessing stored data, namely, the fast path 524 and the slowpath 522. The fast path 524 provides data access when a valid locationID is provided by a client 204 (e.g., within an object handle). The slowpath 522 makes use of the OLS and is used in all other instances of dataaccess. The fast path 524 is so named because a target data object canbe located directly from its (valid) location ID, whereas the slow path522 is so named because it requires a number of additional steps(relative to the fast path) to determine the location of the target dataobject.

The management subsystem 514 includes a content management component 510and an infrastructure management component 512. The infrastructuremanagement component 512 includes logic to allow an administrative userto manage the storage infrastructure (e.g., configuration of nodes,storage devices, volumes, LUNs, etc.).

The content management component 510 is a policy based data managementsubsystem for managing the lifecycle of data objects (and optionally themetadata) stored in the content repository, based on user-specifiedpolicies. It can execute actions to enforce defined policies in responseto system-defined trigger events and/or user-defined trigger events(e.g., attempted creation, deletion, access or migration of an object).

The specified policies may relate to, for example, system performance,data protection and data security. Performance related policies mayrelate to, for example, which logical container a given data objectshould be placed in, migrated from or to, when the data object should bemigrated or deleted, etc. Data protection policies may relate to, forexample, data backup and/or data deletion. Data security policies mayrelate to, for example, when and how data should be encrypted, who hasaccess to particular data, etc. The specified policies can also includepolices for power management, storage efficiency, data retention, anddeletion criteria. The policies can be specified in any known,convenient or desirable format and method. A “policy” in this context isnot necessarily an explicit specification by a user of where to storewhat data, when to move data, etc. Rather, a “policy” can be a set ofspecific rules regarding where to store what, when to migrate data,etc., derived by the system from the end user's SLOs, i.e., a moregeneral specification of the end user's expected performance, dataprotection, security, etc. For example, an administrative user mightsimply specify a range of performance that can be tolerated with respectto a particular parameter and in response the management subsystem 514would identify the appropriate data objects that need to be migrated,where they should get migrated to, and how quickly they need to bemigrated.

FIG. 5B illustrates an example of how the content repository can beimplemented relative to the clustered architecture in FIGS. 2 through 4.Although FIG. 5B illustrates the system relative to a single server node208, it will be recognized that the configuration shown in FIG. 5Bactually can be implemented by two or more (or all) of the server nodes208 in a cluster.

In one embodiment, the distributed object store 518 is implemented byproviding at least one data constituent volume 520 in each of at leasttwo D-modules 216 in the system (any given D-module 216 can include zeroor more single node object stores 520). Also implemented in each of atleast two D-modules 216 in the system are: an OLS store 528 thatcontains mapping data structures used by the OLS 516 including validlocation IDs and policy IDs; and a policy store 526 (e.g., a database)that contains user-specified policies relating to data objects (notethat at least some policies or policy information may also be cached inthe N-module 214 to improve performance).

The presentation layer 502 is implemented at least partially within eachN-module 214. In one embodiment, the OLS 516 is implemented partially bythe N-module 214 and partially by the corresponding M-host 218, asillustrated in FIG. 5B. More specifically, in one embodiment thefunctions of the OLS 516 are implemented by a special daemon in theM-host 218 and by the presentation layer 502 in the N-module 214.

In one embodiment, the management subsystem 514 is implemented at leastpartially within each M-host 218. Nonetheless, in some embodiments, anyof these subsystems may also be implemented at least partially withinother modules. For example, at least a portion of the content managementcomponent 510 of the management subsystem 514 can be implemented withinone or more N-modules 214 to allow, for example, caching of policies insuch N-modules and/or execution/application of policies by suchN-module(s). In that case, the processing logic and state informationfor executing/applying policies may be contained in one or moreN-modules 214, while processing logic and state information for managingpolicies is stored in one or more M-hosts 218. Administrative users canspecify policies for use by the management subsystem 514, via a userinterface provided by the M-host 218 to access the management subsystem514.

As noted above, the distributed object store enables both path-basedaccess to data objects as well as direct access to data objects. Forpurposes of direct access, the distributed object store uses amultilevel object handle, as illustrated in FIG. 5C. When a client 204creates a data object, it receives an object handle 534 as the responseto creating the object. This is similar to a file handle that isreturned when a file is created in a traditional storage system. Thefirst level of the object handle is a system-generated globally uniquenumber, called a global object ID, 537 that is permanently attached tothe created data object. The second level of the object handle is a“hint” which includes the location ID 536 of the data object and, in theillustrated embodiment, the policy ID 538 of the data object. Clients204 can store this object handle 534, containing the global object ID537, location ID 536 and policy ID 538.

When a client 204 attempts to read or write the data object using thedirect access approach, the client includes the object handle of theobject in its read or write request to the server system 202. The serversystem 202 first attempts to use the location ID (within the objecthandle), which is intended to be a pointer to the exact location withina volume where the data object is stored. In the common case, thisoperation succeeds and the object is read/written. This sequence is the“fast path” 524 for I/O (see FIG. 5A).

If, however, an object is moved from one location to another (forexample, from one volume to another), the server system 202 creates anew location ID for the object. In that case, the old location IDbecomes stale (invalid). The client may not be notified that the objecthas been moved or that the location ID is stale and may not receive thenew location ID for the object, at least until the client subsequentlyattempts to access that data object (e.g., by providing an object handlewith an invalid location ID). Or, the client may be notified but may notbe able or configured to accept or understand the notification.

The current mapping from global object ID to location ID is storedreliably in the OLS 516. If, during fast path I/O, the server system 202discovers that the target data object no longer exists at the locationpointed to by the provided location ID, this means that the object musthave been either deleted or moved. Therefore, at that point the serversystem 202 will invoke the OLS 516 to determine the new (valid) locationID for the target object. The server system 202 then uses the newlocation ID to read/write the target object. At the same time, theserver system 202 invalidates the old location ID and returns a newobject handle to the client that contains the unchanged and uniqueglobal object ID, as well as the new location ID. This process enablesclients to transparently adapt to objects that move from one location toanother (for example in response to a change in policy).

An enhancement of this technique is for a client 204 never to have to beconcerned with refreshing the object handle when the location IDchanges. In this case, the server system 202 is responsible for mappingthe unchanging global object id to location ID. This can be doneefficiently by compactly storing the mapping from global object ID tolocation ID in, for example, cache memory of one or more N-modules 214.

As noted above, the distributed object store enables path-based accessto data objects as well, and such path-based access is explained infurther detail in the following sections.

Object Location Transparency Using the Presentation Layer:

In a traditional storage system, a file is represented by a path such as“/u/foo/bar/file.doc”. In this example, “u” is a directory under theroot directory “/”, “foo” is a directory under “u”, and so on.Therefore, a file is uniquely identified by a single path. However,since file handles and directory handles are tied to location in atraditional storage system, an entire path name is tied to a specificlocation (e.g., an inode of the file), making it very difficult to movefiles around without having to rename them.

An inode is a data structure, e.g., a 128-byte structure, which is usedto store information, such as meta-data, about a data container, forexample, a file. The meta-data contained in an inode may include datainformation, e.g., ownership of the file, access permission for thefile, size of the file, file type and location of the file on disk, asdescribed below. The file system uses a file handle, i.e., an identifierthat includes an inode number, to retrieve an inode from a storage disk.

Now refer to FIG. 5D, which illustrates a mechanism that allows theserver system 202 to break the tight relationship between path names andlocation. As illustrated in the example of FIG. 5D, path names of dataobjects in the server system 202 are stored in association with anamespace (e.g., a directory namespace 544). The directory namespace 544maintains a separate directory entry (e.g., 540, 542) for each dataobject stored in the distributed object store 518. A directory entry, asindicated herein, refers to an entry that describes a name of any typeof data object (e.g., directories, files, logical containers of data,etc.). Each directory entry includes a path name (e.g., NAME 1) (i.e., alogical address) of the data object and a pointer (e.g., REDIRECTORPOINTER 1 (shown as stub 1 pointer) for mapping the directory entry tothe data object.

In a traditional storage system, the pointer (e.g., an inode number)directly maps the path name to an inode associated with the data object.On the other hand, in the illustrated embodiment shown in FIG. 5D, thepointer of each data object points to a stub file or a “redirector file”(used interchangeably throughout this specification) associated with thedata object. A redirector file, as indicated herein, refers to a filethat maintains an object locator of the data object. The object locatorof the data object could either be the multilevel object handle 534 orjust the global object ID of the data object. In the illustratedembodiment, the redirector file (e.g., redirector file for dataobject 1) is also stored within the directory namespace 544. In additionto the object locator data, the redirector file may also contain otherdata, such as metadata about the location of the redirector file, etc.

As illustrated in FIG. 5D, for example, the pointer included in thedirectory entry 540 of data object 1 points to a redirector file 545 fordata object 1 (instead of pointing to, for example, the inode of dataobject 1). The directory entry 540 does not include any inode referencesto data object 1. The redirector file for data object 1 includes anobject locator (i.e., the object handle or the global object ID) of dataobject 1. As indicated above, either the object handle or the globalobject ID of a data object is useful for identifying the specificlocation (e.g., a physical address) of the data object within thedistributed object store 518. Accordingly, the server system 202 can mapthe directory entry of each data object to the specific location of thedata object within the distributed object store 518. By using thismapping in conjunction with the OLS 516 (i.e., by mapping the path nameto the global object ID and then mapping the global object ID to thelocation ID), the server system 202 can mimic a traditional file systemhierarchy, while providing the advantage of location independence ofdirectory entries.

By having the directory entry pointer of a data object point to aredirector file (containing the object locator information) instead ofpointing to an actual inode of the data object, the server system 202introduces a layer of indirection between (i.e., provides a logicalseparation of) directory entries and storage locations of the storeddata object. This separation facilitates transparent migration (i.e., adata object can be moved without affecting its name), and moreover, itenables any particular data object to be represented by multiple pathnames, thereby facilitating navigation. In particular, this allows theimplementation of a hierarchical protocol such as NFS on top of anobject store, while at the same time allowing access via a flat objectaddress space (wherein clients directly use the global object ID toaccess objects) and maintaining the ability to do transparent migration.

In one embodiment, instead of using a redirector file for maintainingthe object locator (i.e., the object handle or the global object ID) ofa data object, the server system 202 stores the global object ID of thedata object directly within the directory entry of the data object. Anexample of such an embodiment is depicted in FIG. 5E. In the illustratedexample, the directory entry for data object 1 includes a path name andthe global object ID of data object 1. In a traditional server system,the directory entry would contain a path name and a reference to aninode (e.g., the inode number) of the data object. Instead of storingthe inode reference, the server system 202 stores the global object IDof data object 1 in conjunction with the path name within the directoryentry of data object 1. As explained above, the server system 202 canuse the global object ID of data object 1 to identify the specificlocation of data object 1 within the distributed object store 518. Inthis embodiment, the directory entry includes an object locator (i.e., aglobal object ID) instead of directly pointing to the inode of the dataobject, and therefore still maintains a layer of indirection between thedirectory entry and the physical storage location of the data object. Asindicated above, the global object ID is permanently attached to thedata object and remains unchanged even if the data object is relocatedwithin the distributed object store 518.

Infinite Volume:

FIG. 6 shows an example of an expandable storage volume (may also bereferred to as an Infinite Volume or “InfiniteVol”) 600 that may bepresented to a vserver 608 and used by clients for storing informationwithin the content repository described above, according to oneembodiment. InfiniteVol 600 may include a namespace 602 (similar tonamespace 506 described above with respect to FIGS. 5A-5D), a pluralityof data constituent volumes 604A-604N and an OLS store 606 (similar toOLS store 528 (FIG. 5B). The data constituent volumes are similar to asingle node object stores 520 described above with respect to FIGS.5A-5D. The namespace 602 is used to store the directory 544 or directorynamespace 546, described above in detail. Each data constituent volumemay have its own file system and may be used to store user informationand metadata.

In one embodiment, information regarding the various volumes ofInfiniteVol 600, for example, identifiers for identifying the variousvolumes, vserver identifiers and other information may be stored in adata structure, for example, 220 (FIG. 2) or 708 (FIG. 7A). This allowsa user to expand or contract the size of the InfiniteVol 600 based onuser needs. When a new data constituent volume is added to InfiniteVol600, the data structure 708 is updated to include information regardingthe new volume. When a volume is removed from InfiniteVol 600, then theinformation regarding the volume is removed from the data structure 708.

Data centers typically replicate storage volumes, for example, by taking“snapshots” such that a file system can be restored in case of adisaster. Snapshot (without derogation to any trademark rights ofNetApp, Inc.) means a point in time copy of a storage file system. Asnapshot is a persistent point in time image of an active file systemthat enables quick recovery of data after data has been corrupted, lost,or altered. Snapshots can be created by copying the data atpredetermined point in time to form a consistent image, or virtually byusing a pointer to form the image of the data.

Snapshots for multiple volumes, typically managed by a single node, aretaken by using a “consistency” group. The consistency group is a logicalstructure that includes various storage volumes whose snapshots aretaken at the same time. To take the snapshot of multiple volumes, firstthe consistency group is started and the user is given an opportunity toadd any number of storage volumes. Once the volumes are added, a logical“fence” is generated by the storage operating system 306. The logicalfence is a filtering mechanism that includes the information at thestorage volumes in a snapshot at the time the fence is generated andexcludes any read/write requests that are received after the fence iscreated.

The logical fence may be enabled by the storage operating system 306 bymaintaining a data structure (not shown) at a storage device. The datastructure tracks the I/O requests that are generated after the fence iscreated so that the excluded I/O requests can be handled after thesnapshot is generated. Details of handling the excluded I/O requests arenot germane to the inventive embodiments disclosed herein.

After the snapshot is generated, the storage volumes are “unfenced’ andthe consistency group is deleted. This approach operates well when thestorage volumes are managed by a single node. The process however can bechallenging when one has to take a snapshot for InfiniteVol 600 thatincludes the namespace 602, the OLS store 606 and multiple dataconstituent volumes 604A-604N managed by a plurality of nodes in acluster based environment. The embodiments described herein providesystems and methods for generating snapshots for InfiniteVol 600.

FIG. 7A shows a block diagram of a system 700 for taking snapshots ofInfiniteVol 600, according to one embodiment. System 700 includes a userinterface 702 that may be presented to a client and may include agraphical user interface (GUI) or a command line interface (CLI). Theuser interface 702 may be used to request a snapshot at any given timeor to setup a schedule for automatically taking snapshots at any giventime intervals.

In one embodiment, system 700 includes a snapshot job manager 704 thatis configured to receive a snapshot request for generating a snapshot.The snapshot job manager 704 creates a “job” for taking the snapshot.The snapshot job manager 704 may be implemented by M-host 218. Thesnapshot request is typically received from a client via user interface702. The request may include an identifier (may be referred to as“repository identifier”) that uniquely identifies the InfiniteVol 600and a vserver identifier that uniquely identifies the vserver thatinterfaces with client systems for processing I/O requests and “owns”the InfiniteVol 600. It is noteworthy that the processes and systemsdescribed herein are not limited to using a vserver or a similar entity.A stand-alone storage server may be used to implement the variousembodiments.

The snapshot request is passed by the snapshot job manager 704 to asnapshot coordinator 706 that may also be implemented by M-host 218. Thesnapshot coordinator 706 queries the volume data structure 708 to obtaininformation regarding the various volumes of InfiniteVol 600. The volumedata structure 708 may be maintained by infrastructure management module512 of the management subsystem 514 as described above at a storagelocation accessible to the infrastructure management module 512. In oneembodiment, the volume data structure 708 is a searchable data structurewith fields' 708A-708C shown in FIG. 7B.

Field 708A stores identification information for each vserver in theclustered environment 200 (FIG. 2), for example, Vserver 1. Field 708Bidentifies the InfiniteVols that are presented to each vserveridentified by field 708A, for example, InfiniteVol1. The various volumeswithin each InfiniteVol are identified by field 708C, for example,Namspace1, OLS1 and various data constituent volumes shown as DC1-DCN.As described above, the volumes include namespace 602, data constituentvolumes 604A-604N and OLS store 606.

After the snapshot coordinator 706 obtains information regarding thevolumes of the InfiniteVol 600, a consistency group is started tomaintain consistency across a plurality of nodes 208. An example of aconsistency group 716 is shown in FIG. 7C. The consistency group 716 mayinclude the namespace 718, OLS store 722 and data constituent volumes720A-720N. When the consistency group is started at a given time, alogical “fence” is created first for the namespace 718, then for the OLSstore 722, followed by the data constituent volumes. The term fence asused herein means that the information within each volume, when theconsistency group is started would be included in the snapshot. Thisincludes the completed write requests as well as updated redirectorfiles, namespace and OLS store information. Any read/write operationsafter the consistency group is created are “fenced” off and are notincluded in the snapshot.

After a snapshot is taken, the consistency group goes through a “commit”operation which indicates that the snapshot operation was a success andduring the commit operation the volumes are unfenced. It is noteworthythat even if the snapshot operation is not a success, the volumes arestill unfenced. The order in which the volumes are unfenced is oppositeto the order in which the volumes were fenced. For example, thenamespace 718 is fenced first and unfenced last. The data constituentvolume 720N is fenced last and unfenced first.

One reason the namespace 718 is fenced first is because it includes thedirectory namespace (for example, 544, FIG. 5D) and stub files (forexample, 545, FIG. 5D). The stub files are used to access data objectsthat are stored by the data constituent volumes. If the namespace is notfenced and a data constituent volume is fenced off then there may be amismatch between the namespace entries and the stored data objects atthe data constituent volumes. By fencing the namespace first, one canensure that the data objects stored at that point in time will beconsistent for a given snapshot across the plurality of volumes managedby a plurality of nodes.

The snapshot is taken at a given time across multiple nodes. Once thesnapshot is taken, the storage volumes are unfenced in the orderdescribed above. A snapshot data structure 710 (FIG. 7A) is then updatedor generated if one does not exist. The snapshot data structure 710 isstored at a memory location that is accessible to management subsystem514. An example of the snapshot data structure is shown in FIG. 7Dhaving a plurality of fields, for example, fields 710A-710G that are nowdescribed in detail.

Field 710A identifies the vserver that is associated with theInfiniteVol 600 for which a snapshot is taken at any given time. Field710B identifies the InfiniteVol 600 that is replicated. Field 710Cprovides a name for the overall snapshot and field 710D provides anidentifier for the overall snapshot of InfiniteVol 600. Field 710Eprovides a snapshot identifier for each volume of InfiniteVol 600 thatis replicated and field 710F identifies each volume corresponding to thesnapshot identifier of field 710E. Field 710G provides a timestamp foreach snapshot providing a time when the snapshot was taken.

Snapshot data structure 710 may be used to present snapshots of multiplevolumes across multiple nodes to a user as a single logical entity. Forexample, assume that an InfiniteVol having an identifier, InfiniID1includes 6 volumes, including a namespace identified by Vn1, an OLSstore identified by Vols1 and four data constituent volumes Vd1-Vd4. Thesnapshot for the entire InfiniteVol may be identified as S1 and may berepresented by an object (S1, InfiniD1). The snapshot of the namespacemay be represented as Sn1 and the snapshot of the OLS store may beidentified by Sols1. The snapshots of the four data constituent volumesmay be identified by Sd1, Sd2, Sd3 and Sd4. The overall snapshot may berepresented as:(S1,InfiniD1)=(Sn1,Vn1),(Sols1,Vols1),(Sd1,Vd1),(Sd2,Vd2),(Sd3,Vd3),(Sd4,Vd4).

S1, InfiniD1 may be used to manage the snapshots for various volumes ofan InfiniteVol. S1, InfiniD1 may be presented to a storage administratorusing management console 116 (FIG. 1) for managing the varioussnapshots. A user (for example, client 104) is only presented with asingle object for example, (Sn1, Vn1) to access the various individualsnapshots and is unaware of how the individual snapshots for variousvolumes are being handled by the underlying system. The user is able toaccess the entire snapshot using a single object, as described below inmore detail.

Once the snapshot data structure 710 is updated that information is thenuploaded to a cache 714 of N-Blade 214. In one embodiment, cache 714 maybe used to respond to client requests to access snapshots via a snapshotaccess layer 705 (may also be called a data access layer), as describedbelow in detail.

FIG. 8 shows a process 800 for generating a snapshot for InfiniteVol 600having the namespace 602, the OLS store 606 and the data constituentvolumes 604A-604N managed by a plurality of nodes. The process starts inblock S802 when a request to generate a snapshot of InfiniteVol 600 isreceived. In block S804, snapshot coordinator 706 obtains identifiersfor the volumes within InfiniteVol 600. This includes identifiers forthe namespace, the OLS store and the data constituent volumes. Theidentifier information may be obtained from the volume data structure708 described above with respect to FIG. 7B. Snapshot coordinator 706uses an identifier for the InfiniteVol 600 and an identifier for thevserver as an index into the volume data structure 708 to retrieve thestorage volume information.

In block S806, snapshot coordinator 706 starts a consistency group forthe volumes within InfiniteVol 600 that are to be included in thesnapshot. Namespace 602 is fenced first followed by the OLS store 606and the data constituent volumes 604A-604N. Storage operating system 306(or snapshot coordinator 706) tracks the order in which the volumes arefenced across multiple volumes and multiple nodes. The tracking may beperformed by maintaining a data structure that stores informationregarding namespace 602, OLS store 606 and the data constituent volumes604A-604N. The fence is applied in a serial manner such that the fenceorder mentioned above can be maintained. A time stamp for recording eachfence may also be used for maintaining the order.

The namespace 602 is fenced first because it is presented to users andincludes the directory namespace 544 with directory entries 540 and stubfiles 545. The directory entries 540 store pointers to the stub files545 that point to various objects stored at the data constituent volumes604A-604N. By fencing the namespace 602 first, one can ensure that nochanges to the stub files will be made after the fence is generated.

In block S808, the snapshots for the storage volumes are taken at thesame time. Once the snapshots are taken, the storage volumes are“unfenced” in block S810. The order in which the volumes are unfenced isopposite to the order in which the volumes are fenced in block S806 i.e.the namespace 602 is unfenced last, while the data constituent volumethat was fenced last is unfenced first. By unfencing the namespace 602last, one can ensure that various stub files point to the appropriate,unfenced data constituent volumes. If a data constituent volume isfenced and the namespace 602 is unfenced then a user can submit arequest to read or write an object but the request will not be servicedproperly because the data constituent volume is still fenced when therequest is received and therefore, an error may be generated.

Storage operating system 306 (or snapshot coordinator 706) maintains arecord (not shown) of the order in which storage volumes are fenced.This information may be stored at any storage location that isaccessible to snapshot coordinator 706 or any other module that canprovide this information to snapshot coordinator 706.

Thereafter, in block S810, snapshot coordinator 706 updates the snapshotdata structure 710 that has been described above in detail with respectto FIG. 7D. The snapshot can then be presented as a single entity to auser. The snapshot has its own name and identifier that can be used toaccess the snapshots of the underlying storage volumes, as describedbelow in more detail. Thereafter, the process ends in block S812.

FIG. 9A shows a process 900 for accessing a snapshot of the InfiniteVol600 taken by the process described above with respect to FIG. 8. Theprocess begins in block S902, when a handle to access the snapshot isreceived by the snapshot access layer 705. The handle includes asnapshot identifier for the snapshot of the namespace and a namespaceidentifier, for example, Sn1, Vn1, as described above.

In block S904, the snapshot access layer 705 retrieves a dataconstituent volume identifier and a snapshot identifier for the dataconstituent volume using the namespace identifier and the snapshotidentifier from the snapshot data structure 710 that has been describedabove.

In block S906, an active file system identifier (as stored in thestub/redirector file 545) is replaced by the data constituent snapshotidentifier and a handle is provided to the D-module 216 that manages thedata constituent volume and its snapshot. Thereafter, access to thesnapshot of the data constituent volume is provided to the user.

An example of implementing process 900 is shown in FIG. 9B. The initialfile handle (or client request) 908 includes a snapshot identifier 4, anamespace identifier 100 and an inode identifier 2000. The namespaceidentifier points to a stub file 910 that includes a data constituentvolume identifier 200 with an inode identifier 3000. The active filesystem value, which may be 0 is replaced by 3, the snapshot identifierfor data constituent volume 200. The snapshot identifier value of 3 isobtained by using the various fields of snapshot data structure 710described above.

Data handle 910 with the snapshot identifier 3, volume identifier 200and inode number 3000 is provided to the D-module 216 that manages thedata constituent volume 200. The D-module 216 then returns the snapshotdata that is provided to the client.

In one embodiment, clients within a content repository are provided withan expandable storage volume having a plurality of volumes that may bemanaged by different storage server nodes. The plurality of volumes isreplicated using the techniques described above. The clients can use asingle object to access each replicated volume without having to spendany resources in managing the replicated volumes.

The techniques introduced above can be implemented by programmablecircuitry programmed or configured by software and/or firmware, orentirely by special-purpose circuitry, or in a combination of suchforms. Such special-purpose circuitry (if any) can be in the form of,for example, one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field-programmable gatearrays (FPGAs), etc.

Software or firmware for implementing the techniques introduced here maybe stored on a non-transitory, machine-readable storage medium and maybe executed by one or more general-purpose or special-purposeprogrammable microprocessors. A “machine-readable medium”, as the termis used herein, includes any mechanism that can store information in aform accessible by a machine (a machine may be, for example, a computer,network device, cellular phone, personal digital assistant (PDA),manufacturing tool, any device with one or more processors, etc.). Forexample, a machine-accessible medium includes recordable/non-recordablemedia (e.g., read-only memory (ROM); random access memory (RAM);magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc.

The term “logic”, as used herein, can include, for example,special-purpose hardwired circuitry, software and/or firmware inconjunction with programmable circuitry, or a combination thereof.

Cloud Computing:

The system and techniques described above are applicable and useful inthe upcoming cloud computing environment. Cloud computing meanscomputing capability that provides an abstraction between the computingresource and its underlying technical architecture (e.g., servers,storage, networks), enabling convenient, on-demand network access to ashared pool of configurable computing resources that can be rapidlyprovisioned and released with minimal management effort or serviceprovider interaction. The term “cloud” is intended to refer to theInternet and cloud computing allows shared resources, for example,software and information to be available, on-demand, like a publicutility.

Typical cloud computing providers deliver common business applicationsonline which are accessed from another web service or software like aweb browser, while the software and data are stored remotely on servers.The cloud computing architecture uses a layered approach for providingapplication services. A first layer is an application layer that isexecuted at client computers. In this example, the application allows aclient to access storage via a cloud.

After the application layer, is a cloud platform and cloudinfrastructure, followed by a “server” layer that includes hardware andcomputer software designed for cloud specific services. The managementconsole 118 (and associated methods thereof) and storage systemsdescribed above can be a part of the server layer for providing storageservices. Details regarding these layers are not germane to theinventive embodiments.

Thus, a method and apparatus for replicating an expandable storagevolume have been described. Note that references throughout thisspecification to “one embodiment” or “an embodiment” mean that aparticular feature, structure or characteristic described in connectionwith the embodiment is included in at least one embodiment of thepresent invention. Therefore, it is emphasized and should be appreciatedthat two or more references to “an embodiment” or “one embodiment” or“an alternative embodiment” in various portions of this specificationare not necessarily all referring to the same embodiment. Furthermore,the particular features, structures or characteristics being referred tomay be combined as suitable in one or more embodiments of the invention,as will be recognized by those of ordinary skill in the art.

While the present disclosure is described above with respect to what iscurrently considered its preferred embodiments, it is to be understoodthat the disclosure is not limited to that described above. To thecontrary, the disclosure is intended to cover various modifications andequivalent arrangements within the spirit and scope of the appendedclaims.

What is claimed is:
 1. A machine implemented method for generating asnapshot of an expandable storage volume in a network storage system,the expandable storage volume having a namespace for storing informationfor accessing data objects stored at one or more data constituentvolumes, comprising: initiating a logical fence first for the namespaceand then for an object location subsystem (OLS) datastore to track dataobjects when moved within the one or more data constituent volumes andafter that for the one or more data constituent volumes for generatingthe snapshot of the expandable storage volume, wherein the logical fenceexcludes any write request to write within the logical fence after thelogical fence is created; storing information regarding the snapshot ofthe expandable storage volume in a data structure such that the snapshotof the expandable storage volume is presented to the client as a singleentity for accessing a snapshot of the namespace and snapshots of theone or more data constituent volumes; after the snapshot of theexpandable storage volume is generated, removing the logical fence in anorder opposite to an order in which the fence was initiated; receivingan object for accessing the snapshot of the expandable storage volume,the object including an identifier identifying the snapshot of thenamespace and an identifier identifying the namespace; retrieving anidentifier identifying at least one of the snapshots of the one or moredata constituent volume from the data structure; and retrieving dataassociated with the snapshot of the data constituent volume based on theidentifier of the snapshot of the data constituent volume.
 2. The methodof claim 1, wherein the data structure stores an identifier of thesnapshot of the expandable storage volume, the identifier identifyingthe snapshot of the namespace, the identifier of the at least onesnapshot of the one or more data constituent volumes, the identifier ofthe namespace and an identifier identifying each of the one or more dataconstituent volumes.
 3. The method of claim 2, wherein the identifier ofthe namespace is used to obtain the identifier of one of the dataconstituent volumes and the identifier of the snapshot of the namespaceis used to obtain the identifier of the snapshot of the one of the oneor more data constituent volumes.
 4. The method of claim 1, wherein theexpandable storage volume includes a plurality of data constituentvolumes managed by at least two different storage system nodes.
 5. Themethod of claim 1, further comprising: retrieving identifier informationof the namespace and the one or more data constituent volumes using anidentifier of the expandable storage volume from a volume data structuremaintained by a storage system node.
 6. The method of claim 1, whereinthe namespace stores a directory namespace having a directory entryassociated with a data object stored at the one or more data constituentvolumes.
 7. The method of claim 6, wherein the directory entry stores apointer to a redirector file that includes an object locator of the dataobject.
 8. A system comprising: a memory containing machine readablemedium comprising machine executable code having stored thereoninstructions; and a processor module coupled to the memory, theprocessor module configured to execute the machine executable code to:initiate a logical fence first for a namespace and then for an objectlocation subsystem (OLS) datastore to track data objects when movedwithin one or more data constituent volumes and after that for the oneor more data constituent volumes for generating a snapshot of anexpandable storage volume in a network storage system having thenamespace for storing information for accessing data objects stored atthe one or more data constituent volumes, wherein the logical fenceexcludes any write request to write within the logical fence after thelogical fence is created; store information regarding the snapshot ofthe expandable storage volume in a data structure such that the snapshotof the expandable storage volume is presented to the client as a singleentity for accessing a snapshot of the namespace and snapshots of theone or more data constituent volumes; remove the logical fence in anorder opposite to an order in which the fence was initiated, after thesnapshot of the expandable storage volume is generated; receive anobject for accessing the snapshot of the expandable storage volume, theobject including an identifier identifying the snapshot of the namespaceand an identifier identifying the namespace; retrieve an identifieridentifying at least one of the snapshots of the one or more dataconstituent volume from the data structure; and retrieve data associatedwith the snapshot of the data constituent volume based on the identifierof the snapshot of the data constituent volume.
 9. The system of claim 8wherein the data structure stores an identifier of the snapshot of theexpandable storage volume, the identifier identifying the snapshot ofthe namespace, the identifier of the at least one snapshot of the one ormore data constituent volumes, the identifier of the namespace and anidentifier identifying each of the one or more data constituent volumes.10. The system of claim 9 wherein the identifier of the namespace isused to obtain the identifier of one of the data constituent volumes andthe identifier of the snapshot of the namespace is used to obtain theidentifier of the snapshot of the one of the one or more dataconstituent volumes.
 11. The system of claim 8 wherein the expandablestorage volume includes a plurality of data constituent volumes managedby at least two different storage system nodes.
 12. The system of claim8 further comprising code to: retrieve identifier information of thenamespace and the one or more data constituent volumes using anidentifier of the expandable storage volume from a volume data structuremaintained by a storage system node.
 13. The system of claim 8 whereinthe namespace stores a directory namespace having a directory entryassociated with a data object stored at the one or more data constituentvolumes.
 14. A non-transitory, machine readable storage medium havingstored thereon instructions for performing a method, comprising machineexecutable code which when executed by at least one machine, causes themachine to: initiate a logical fence first for a namespace and then foran object location subsystem (OLS) datastore to track data objects whenmoved within one or more data constituent volumes and after that for theone or more data constituent volumes for generating a snapshot of anexpandable storage volume in the network storage system having thenamespace for storing information for accessing data objects stored atthe one or more data constituent volumes, wherein the logical fenceexcludes any write request to write within the logical fence after thelogical fence is created; store information regarding the snapshot ofthe expandable storage volume in a data structure such that the snapshotof the expandable storage volume is presented to the client as a singleentity for accessing a snapshot of the namespace and a-snapshots of theone or more data constituent volumes; remove the logical fence in anorder opposite to an order in which the fence was initiated, after thesnapshot of the expandable storage volume is generated; receive anobject for accessing the snapshot of the expandable storage volume, theobject including an identifier identifying the snapshot of the namespaceand an identifier identifying the namespace; retrieve an identifieridentifying at least one of the snapshots of the one or more dataconstituent volume from the data structure; and retrieve data associatedwith the snapshot of the data constituent volume based on the identifierof the snapshot of the data constituent volume.
 15. The storage mediumof claim 14 wherein the code causes the data structure stores anidentifier of the snapshot of the expandable storage volume, theidentifier identifying the snapshot of the namespace, the identifier ofthe at least one snapshot of the one or more data constituent volumes,the identifier of the namespace and an identifier identifying each ofthe one or more data constituent volumes.
 16. The storage medium ofclaim 15 wherein the identifier of the namespace is used to obtain theidentifier of one of the data constituent volumes and the identifier ofthe snapshot of the namespace is used to obtain the identifier of thesnapshot of the one of the one or more data constituent volumes.
 17. Thestorage medium of claim 14 wherein the expandable storage volumeincludes a plurality of data constituent volumes managed by at least twodifferent storage system nodes.
 18. The storage medium of claim 14further comprising instructions to: retrieve identifier information ofthe namespace and the one or more data constituent volumes using anidentifier of the expandable storage volume from a volume data structuremaintained by a storage system node.
 19. The storage medium of claim 14wherein the namespace stores a directory namespace having a directoryentry associated with a data object stored at the one or more dataconstituent volumes.